Summary

• preserve complete real-user token deltas when Qwen3 rewrites earlier reasoning
• preserve the assistant generation prefix on every later assistant turn
• mask generation-prefix tokens with 0 and generated continuation tokens with 1
• handle legal BPE merges across the generation-prefix/continuation boundary
• fail closed on unexpected non-user history rewrites

Root cause

The tau-bench example used curr[len(prev):] for every newly appended message. Qwen3 violates that assumption in two ways:

1. After a new real user arrives, its official chat template removes older assistant reasoning, so curr no longer starts with prev; slicing by the old length can truncate the new user.
2. The first assistant generation prefix is already in the initial prompt, but later prefixes were never appended to the accumulated GRPO token stream. Later turns could therefore miss <|im_start|>assistant\n<think>.

A further tokenizer boundary case matters: a continuation beginning with whitespace can BPE-merge with the end of the generation prefix, so separately tokenized prefix IDs are not guaranteed to be an exact token prefix of the combined assistant span.

Fix

• Keep the append-only fast path when history is stable.
• If a new real user rewrites history, render that user independently and verify it is the exact rendered suffix.
• For later assistant turns, append the tokenizer-rendered generation prefix plus continuation; mask the prefix with 0 and the continuation with 1.
• Derive the masked prefix length without requiring separate-prefix token IDs to equal the combined span prefix.

This retains the existing tau-bench update convention: accumulated GRPO training tokens keep prior reasoning. The change fixes token construction and masks; it does not change rollout visibility semantics.

Validation

• pytest -q tests/test_tau_bench_token_delta.py — 4 passed
• synthetic history-rewrite, missing-assistant-prefix, and BPE-boundary regressions
• real tokenizer parity against Slime MultiTurnLossMaskGenerator(..., tokenizer_type="qwen3"):
  • Qwen3-4B-Instruct-2507: normal and leading-newline continuations
  • strict Epoch-4 Qwen3-4B Thinking checkpoint: normal and leading-newline continuations
  • token IDs and masks match exactly in all four cases